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ABSTRACT 

This study addressed research collaboration on a 
range of instructional and administrative issues concerning how best 
to serve the educational needs of students with disabilities or at 
risk in general education classrooms. A group of expert researchers 
(N=31) , funded to conduct research on the General Education 
Initiative (GEl) , was interviewed to examine the research variables 
they selected and their procedures for implementing them. The 
selected research variables were arrayed against a framework of 
variables found to be influential in learning. Issues concerning the 
procedures that researchers follow to select their methods for 
operationalizing the variables were considered. Needs for technical 
assistance in this area were presented. Finally, patterns of 
collaboration and suggestions to facilitate additional collaboration 
among researchers were elicited. The study found a lack of research 
on parental factors related to the GEI and on the issues of common 
demographic and marker variables or instrumentation. Appended are the 
interview protocol and a list of influential variables in learning 
addressed by 28 federally funded projects. (Contains 14 references.) 
(DB) 
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ABSTRACT 



Research integration in fields in which research is prolific continues to 
be a problem. In this study a group of expert researchers funded to conduct 
research on the GEI were interviewed to examine the variables they selected 
and their procedures for operationalizing them. The variables were arrayed 
against a framework of variables found to be influential in learning. Issues 
concerning the procedures researchers follow to select their methods for 
operationalizing the variables were considered. Needs for technical assistance 
in this area are presented. Finally, patterns of collaboration and suggestions 
to facilitate additional collaboration among researchers were elicited. 
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OPERATIONALIZING THE MARKER-VARIABLE SYSTEM: 
RESEARCHERS’ SELECTION OF MEASUREMENT TECHNIQUES 

Much of the literature in any scientific field is contributed by relatively 
isolated investigators, and the resulting research often consists of a multitude 
of uncoordinated studies. Attempts then are made to integrate the field using 
techniques of synthesis such as literature reviews and meta-analyses. However, 
the synthesis is impeded by the limited scope of the individual studies and of 
their samples, and the difficulties in aligning the studies either conceptually or 
methodologically (Bell & Hertz, 1976). According to Bell and Hertz, the 
problem of integrating research on a given topic particularly needs to be 
addressed when there is a "quickening of interest in the field, indicating that 
a large number of studies will soon be carried out, and yet sufficient work has 
already been done so as to make it possible to establish . . . empirical 
anchors” (p. 10). 

One useful strategy to facilitate integration in such circumstances is to 
identify marker variables (Bell & Hertz, 1976). A marker variable is defined as 
"a background variable (not necessarily the focal variable in the study) that is 
sufficiently relevant to the measures being used by most studies in a defined 
research area that it facilitates the general alignment of findings from one 
study to another" (Bell & Hertz, 1976, pp. 8*9). While more substantive than 
demographic variables, both types of variables are useful to those who attempt 
to synthesize findings across multiple studies in a given field. 



It is recommended by Bell and Hertz that agencies, such as federal 
offices that support research, would serve the progress of a field of knowledge 
if they would both encourage and respond to efforts by investigators to join 
together in achieving better definition of measures and samples and to 
facilitate collaboration in planning and execution of studies. They believe that 
support of efforts for collaboration is especially important when "early efforts 
have burgeoned into a plethora of uncoordinated constructs, techniques, and 
measures" (p. 11). 
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This study addresses research collaboration with particular respect to the 
literature addressing how best to serve the educational needs of at risk and 
handicapped students. A series of interviews was conducted with senior 
researchers who were funded to explore the initiative of the Office of Special 
Education Programs, U.S. Department of Education, concerning a range of 
instructional and administrative issues in educating learning disabled and mildly 
handicapped students in general education classrooms. The variables addressed 
by the researchers were identified and considered in light of a framework of 
variables important to the learning process. In addition, operationalization 
strategies of these expert researchers were elicited, and responses related to 
stimulating and facilitating collaboration among researchers were obtained. 

In the past, there have been some attempts to cope with a field of study 
in which the literature was growing and synthesis was difficult to build. 
Within the field of child development, for example, there was an attempt to 
develop methods of aligning studies in intervention with disadvantaged children 
in the first three years of life. The problem of specifying sample 

characteristics in a uniform system for all studies was addressed by a small 
group of investigators, who began by exchanging reports on research results 
and then turr * |. to develop a uniform set of sample characteristics. A formal 
project (Gordon, Beller, Lally, Moreno, Rand, & Freiberg, 1973) was supported 
by the Office of Child Development to standardize methods of measuring socio- 
emotional variables for this group. Bell and Hertz (1976), in tracing some of 
the roots of previous attempts to develop marker variable systems, describe 
two instances in which a group of investigators came together; that of 
cognitive research (French, Elstrom, & Price, 1963; French, 1973) and of the 
psychophysiology of sleep (Rechtschaffen & Kales, 1968). 

A further example of coordination across studies, in this case provided by 
the U.S. Department of Education, is the first-grade reading study project 
(Bond & Dykstra, 1967). In this case the U.S. Department of Education funded 
multiple studies of first-grade reading instruction, including several distinct 
approaches, each of them explicated at various sites across the nation. In 
addition, the Department funded a special project for coordination of the 
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studies under the leadership of Professor Bond and Dykstra at the University 
of Minnesota. Lead researchers of all projects were convened by Bond and 
Dykstra. Working committees were formed to seek agreement on efforts for 
coordination, such as tests that would be used in common across projects to 
measure instructional outcomes. Procedures for general analysis of data 
(across projects) were also established. For example, a broad study seeking to 
identify ATI’s (Aptitude Test Interactions) was planned and eventually 
executed. This effort involved a large number of the nation’s top researchers 
in the field of reading instruction and was carried through with good 
cooperation and notably good results. It involved all of the researchers in 
planning and carrying out the coordination. It was definitely not a top down 
operation, except in the sense that the staff of the Federal sponsoring agency 
was aware of the need and opportunity for coordination and offered support 
for the meetings and leadership required to carry through the entire effort. 

A data base has been established at MIT into which researchers in child 
language may deposit their own data and withdraw data from other 
researchers. According to Butler (personal communication): 

A variety of systems for storing the data in accessible forms (thus 
also requiring that data be provided to that system in specific 
coding systems) has permitted researchers in child language (who 
typically deal in very small N’s) to draw from much larger pools of 
similarly gathered data. A massive study of more than 500 subjects 
is now being entered into the data base, which combines the 
resources of researchers at the Harvard Graduate School of 
Education, those at MIT, and other university-based researchers; all 
at little or no cost to individual researchers. The MIT program is 
working well . . . the individual researchers may use combined data 
in a variety of ways, none are coerced to either provide or utilize 
the data, there are systems which permit multiple uses, and yet the 
data base is constrained in manageable ways. Such a "marker 
system," if you will, demonstrates great potential for collaborative 
research or for individual use of well-gathered data by others. 




6 



Directly related to the content area of this research, Keogh and her 
colleagues (Keogh, Major-Kingsley, Omori-Gordon, & Reid, 1982) reported an 
attempt in the field of learning disabilities to encourage cooperation and 
coordination among researchers. In attempting to synthesize the research on 
learning disabilities, it became clear that one of the major problems was in the 
way that the research was reported. The lack of reporting on basic 
demographic variables and other factors made it difficult to align the studies. 
The UCLA Marker Variable Guide (Keogh, et al, 1982) was developed as a list 
of variables or markers to describe samples of subjects in learning disabilities 
research. The indicators were not to be used as an attempt to limit what 
research is done or reported, but rather to facilitate "commonality of reporting 
that will allow determination of samples equivalence" (Keogh et al, 1982, p. 
81). Although published in 1982, the Guide has seen little implementation. 

More recently, on February 26, 1989, the National Joint Committee on 
Learning Disabilities produced a statement indicating their concern about the 
lack of consistency in describing subjects in learning disability research, and 
provided a list of subject descriptors which they recommended for use in 
learning disabilities research. International efforts have also been directed 
towards facilitating syntheses across countries. There is increasing movement, 
in part sponsored by the United Nations, towards utilizing the same 
instruments or comparable approaches in measuring mathematics and science 
achievement across nations (H. Walberg, personal communication). 

Current efforts to develop alternatives to educate the mildly handicapped 
are in a state in which collaboration among researchers and the development 
of a marker variable system could be beneficial. The so-called GEI (General 
Education Initiative), or shared responsibility position, conforms to Bell and 
Hertz’s indicator for the ideal time for a marker variable system, i.e., where 
there has been a quickening of interest in the field, a large number of studies 
either in progress or being planned, and sufficient work having already been 
done, making it possible to establish the relevant empirical anchors. In 
recognition of this state of affairs, this project was funded by the Office of 
Special Education and Rehabilitative Services to conduct such a process: to 
synthesize the current state of research and practice, and provide a synthesis 
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of the knowledge base (see Deliverable D, "A Decision-Making Framework for 
Description of Innovative Education Programs"). 

Certain assumptions underlie the study, as they relate to methodology. It 
is assumed that perceived cumulativeness in a research domain is in part a 
function of certain conventions of evidence and methodology shared by the 
research community. As Hedges (1987) suggests: "the study of relative 

cumulativeness across research domains becomes (at least in part) a study of 
conventions used by the research community for achieving a sense of 
cumulativeness” (p. 453). As we look for research to inform policy, it is 
important that inconsistent research results not be primarily due to differences 
in methodology. While there is no implication here that researchers' 
independence or freedom to design or operationalize their research should be 
constrained, the question is being asked if there is some responsibility to look 
at research procedures to determine where collaboration and agreement might 
be possible to more readily achieve comparable findings. 

Moreover, there is increasing interest in the way that given professionals 
think about their work. From Schoen’s (1983) work on the reflective 
practitioner to work on teachers’ thinking (e.g., Clark & Peterson, 1986), it is 
a reasonable move to examine the thinking processes of researchers. 
Examining how expert researchers think about and plan their research might 
enable us to learn something about this process that would be useful in the 
teaching of new researchers, as well as in knowing how we might improve the 
cumulativeness of research in a given field. 

A marker variable system (Reynolds, Wang, & Walberg, 1989) has been 
developed from an extensive and systematic review of the literature. This 
comprehensive set of variables has been organized into an inclusive conceptual 
framework of six categories and 228 items. These have been presented to a 
wide range of stakeholders responsible for research and practice in education. 
In addition, each item has been rated for the degree of empirical support 
found for it in the literature (Wang, Haertel, & Walberg, 1989). 

The second step in this synthesis is the focus of this study, whose 
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purpose is to examine which of these variables have been utilized in the 
burgeoning research on the General Education Initiative, and the 
instrumentation issues involved in their operationalization. Several related 
questions were asked: 

1. Which variables identified as influential in learning have been included in 
the OSER-funded research on the GEI? 

2. How have the variables been operationalized in terms of measurement 
procedures and instruments? 

3. What factors influenced the selection of measurement procedures? 

4. How might collaboration among researchers on variable selection and 
instrumentation be facilitated? 

5. What technical assistance needs are common among researchers in the 
field regarding instrumentation/operationalization of these influential 
variables? 



Method 



Subjects 

A list of 34 projects, funded for their first year between 1985 and 1988 
by the Office of Special Education and Rehabilitative Services, to develop a 
knowledge base for the GEI, was obtained from the funding agency. The 
subjects of this study are the Principal Investigators (Pi’s), or their designated 
representative(s), who were the recipients of the grants in these competitions. 
The subjects for this study represent 29 of the 34 projects. A total of 31 
individuals were interviewed; for four of the projects, two individuals were 
interviewed jointly. Three of the individuals interviewed were each funded for 
two projects. For a variety of scheduling and other reasons, it was not 
possible to interview anyone from five of the projects. 

Procedures 

The design of the study involved interview data, collected by telephone, 
using a structured interview protocol. A list of names, phone numbers, and 
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project abstracts for each research grant funded under the GEI was obtained 
from OSERS. The original intent had been to conduct much of the process 
through an evaluation of the grant proposals of the funded projects. However, 
informal contact with several of the Pi’s indicated that there had been 
multiple changes from the original submissions. A pilot project was conducted 
using written materials and phone contact during the summer of 1988, and 
presented at a meeting in Washington, DC, conducted by the Center for 
Research in Human Development and Education staff of this project. Based on 
feedback from those attending this meeting and other members of the project 
advisory board, the structured interview protocol was developed. 

The interview protocol (found in Appendix A) focused on questions 
related to the major variables and measurement methodologies utilized. The 
following were identified for each funded research study: 

1. Demographic variables about the sample. 

2. Independent and dependent variables. 

3. Research procedures for collecting data on each variable;. 

4. The reason for selection of the measurement procedure or 

instrumentation. 

5. Strengths and limitations of the instruments. 

In addition, other questions were asked related to: 

6. Interest/experience of the researchers in collaboration with other 

researchers, as well as ideas about facilitating research collaboration. 

7. Interest and need for additional resources/technical assistance with 

respect to instrumentation to measure the variables. 

An introductory letter was sent to each PI, explaining the nature of the 
project and requesting participation. (A copy of the letter sent during the 
second phase of interviews is included in Appendix A). The letter was followed 
by a phone call to the Pi’s, confirming their interest in participating in the 
interview process, inviting them to designate themselves or a senior researcher 
on the project for that purpose, as well as setting a time convenient to 
interviewers and interviewees. The interviews were designed as structured 
conversations between researchers, i.e., interviewer and interviewee, and were 
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conducted in a conversational format between colleagues about research. 

Three research associates conducted the interviews. One interview was 
conducted to pilot the interview protocol, and two additional interviews were 
conducted jointly by the interviewers to promote standardization of the format. 
The remaining interviews were conducted individually. The interviewers 
completed the written protocol, which was then coded for analysis. 



Research Variables 

The first question addressed which variables identified as influential in 
learning have been included in the OSER-funded research on the GEI. Each 
demographic, independent and dependent variable was identified and coded for 
28 projects (the results of one project interview were received too late to be 
included in this analysis) according to the Marker Variable Framework 
described by Reynolds, Wang, and Walberg (1989). When a variable did not fit 
the original framework, it was coded as a new item. All variables were coded 
within the S'? a." categories used in the framework, but 23 new items were 
developed. Appendix B consists of a copy of the complete framework, so that 
the number of variables not included as well as those found in these studies 
can be seen. The mean number of variables per study was 11.3; the range 
varied from 2 to 24. Data from five interviews were coded, separated by two 
raters to determine reliability. A coder reliability of .88 was obtained. 

Demographic Variables 

One of the major concerns in development of a marker variable system 
for research is the presentation of consistent background and demographic data 
in all studies relevant to the same topic. In the 28 projects analyzed, the 
mean number of demographic variables was 3.07, with a range of 0 to 14 
demographic variables collected by the researchers. 



Results 
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However, the specific demographic data gathered varied widely. The 
highest consistency is over relatively few categories, including: type of 

district, i.e., suburban/rural/urban; type of school, i.e., public/private; special 
education classification; grade level; socio-economic status; gender; and 
ethnicity. Only three (gender, classification, and grade level) were found in 
half or more of the studies. The demographic information generally was 
collected from school records, and some researchers expressed some concern 
about accuracy of data. 

In all but one study the school system’s special education classification 
of the student was accepted, and it was unclear if the basis for the 
classification would be presented to readers of the research. Given the 
documented variability of classification systems across schools, there is cause 
for concern when we come to ask what type of population a program has been 
demonstrated to serve. 

Variables bv Category 

A second question involves which of the variables in the Marker Variable 
System have been pinpointed for study by these funded grants. The frequeny 
count for variables will be presented in terms of the 6 categories and 30 
subcategories, rather than the 228 specific items (although these data are 
available in Appendix B). None of the studies included variables from category 
1, State and District Variables; 15 of 28 included variables in category 2, Out 
of School Contextual Variables. Of these, 12 studies only included Community 
Variables (consisting almost entirely of the Suburban, Rural, Urban item), 
although three had variables in the Home Environment and Parental Support 
Subcategory. In addition, the research studies examined variables in the other 
categories as follows: 22 of 28 studies for category 3, School Level Variables; 
27 of 28 for category 4, Student Variables; 6 of 28 for category 5, Program 
Design Variables; and 17 of 28 for category 6, Implementation, Classroom 
Instruction, and Climate Variables. Currently, school and student variables are 
the major focus of the research, which is appropriate given the nature of the 
GEI initiative under which these grants were funded. 
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Instrumentation 

Specific to this study are the questions relating to measurement and 
instrumentation. The data were arrayed to determine if the same variable is 
being assessed in different ways by the different researchers. Table 1 shows 
that 72 variables were assessed by researcher-created instruments, in 
comparison to 48 variables assessed through published instruments. Tables 2 
and 3 indicate the number of different achievement and observation measures 
in use in these studies. Table 4 shows the number of times individual 
instruments were used across studies. Academic achievement demonstrates this 
variety. Reading achievement, a common variable in the studies, was measured 
by group achievement tests, including: CTBS, Stanford Achievement, 

Metropolitan, Iowa Test of Basic Skills, and the California Achievement Test. 
Individual tests were also used, including the WRAT and the Woodcock- 
Johnson, as well as curriculum-based assessment and curriculum-based 
measurement techniques. In addition, some studies used grades or teacher 
rankings. 

A perceived need to examine the data more qualitatively was clear among 
these researchers, many of whom collected some qualitative data. Given the 
recent interest in qualitative data (see also the interest in qualitative data as 
a topic for technical assistance conferences, listed in Table 6), it is hard to 
know if the use of this method reflects the current research "Zeitgeist", the 
problems of this area of research or some combination of the two. It should be 
noted that most of the research studies used multiple data collection sources 
for the different variables. Classroom observation, interview data, 
questionnaires, standardized and curriculum based assessment/measurement 
techniques were often found in the same study. 

The reasons for selecting the instrumentation methodology were also 
obtained. Table 5 rank orders the reasons for the selection of methodology 
for 18 Pi’s selection of 125 measurement strategies. Some researchers reported 
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one reason for a selection, while giving multiple reasons for others. The two 
most frequent reasons, ease of use and the need to create the technique for 
the study because nothing else was available suggest both the autonomy of the 
researchers and the difficulty of doing research in the schools. 

Technical Assistance 

The Pi’s were asked to suggest the kinds of technical assistance they 
would have found helpful. Table 6 displays the results of that question. The 
need for technical assistance around qualitative methodology was apparent. In 
addition, there was an interest in simply having the opportunity to share ideas 
and problems, as well as techniques. 

Collaboration 



Finally, questions were raised about the level of collaboration among 
researchers. Table 7 demonstrates that considerable collaboration is already 
ongoing among these researchers. While joint data collection and instrument 
sharing is less common, as a group they were interested in more opportunities 
to share and provided numerous suggestions to facilitate that process. Table 8 
documents these suggestions. Several of the suggestions could be implemented 
by the funding agencies when research grants on a common problem are to be 
funded. Table 9 includes some additional issues and concerns raised by the 
researchers. 



Discussion 

The results of these interviews with this expert group of researchers 
raise a number of implications and policy considerations. Using the Marker 
Variable System as a framework, it is possible to evaluate which variables 
related to learning have received more versus less attention for a given 
population or problem. It would appear, for example, that little work has 

been done so far around parental factors related to the GEI. Secondly, the 
issue of common demographic and marker variables has not yet been addressed 
in the special education research literature, in spite of the apparent need for 
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such consensus and the pioneering work done by Keogh early in the 1980’s. 
Reaching consensus on some common variables across studies, as well as 
clarifying how to define special education and at risk populations across 
studies, rather than relying on definition by individual idiosyncratic school 
classification systems, is likely to benefit the development of research 
integration and has been attempted in other research areas. 

Issues of instrumentation also need further consideration. Studies of the 
comparability of techniques presuming to measure the same variable would 
resolve some of the problem. Perhaps some conventions among researchers 
regarding the nature of and conditions in which variables such as time-on- 
task are measured would improve the literature considerably. Are there some 
variables that need to be "standardized" in terms of common meanings and 
comparable measurement procedures? Because of the energy and commitment 
of several individuals, some movement toward joint instrument use and shared 
data collection has begun among this group of funded projects. Two of the 
interviewees also mentioned consortia around research outside this particular 
group of projects. Collaboration within schools has become an emerging theme 
of the school reform movement, and perhaps the interest displayed by the 
researchers in further collaboration is a reflection of the times as well as 
their recognition of the benefits to be obtained. It might also be useful to 
explore how other such groups of researchers managed to reach consensus 
regarding meaning and measurement operations. 

Many of the reasons given for the selection of the type of 
instrumentation reflected the "5 minutes, 5 cents rule": the measurement needs 
to be easy to administer, economical, and brief. This is a realistic concern. 
The researchers were all working in the real world (how researchers enter 
schools and districts is another issue worth investigating) and were working 
with school personnel whose main focus was not research. Although these 
concerns are realistic, they do not invalidate the need to monitor that the 
purpose or goal of the research does not become subordinate to feasibility 
issues. This is a difficult line, requiring considerable researcher skill. The 
need to develop school/research collaboration is clearly illuminated by these 
researchers’ difficulties in gaining cooperation. Strategies such as small gifts. 





using instrumentation that would provide useful feedback to the school 
personnel, and ease and speed of administration were strategies described by 
the researchers. Most of these researchers acknowledged how difficult it was 
to try to do good research within the confines of their funding and the 
problems in gaining cooperation from the schools. Collecting data for applied 
research in field situations is a difficult process, in which measurement issues 
sometimes rank second to the realities of the school culture and need to be 
carefully balanced by experienced researchers. Perhaps we need to consider 
how to make the schools more of a joint collaborator in the research process. 
Porter (1987) describes a pioneering collaboration effort between university 
researchers and teachers, which suggests integrating practitioners more directly 
and at an earlier point into research projects. 

The researchers also made a number of suggestions to funding agencies to 
encourage a culture of support among researchers. Several suggested providing 
opportunities to collaborate early in the projects, through meeting and through 
sharing information and instruments. They recommended using major 
conferences as opportunities for additional meetings. They asked for 
incentives to collaborate, rather than compete. Given the competition inherent 
in the grant process, these researchers recognized the need to build a climate 
of trust. The development of research networks was seen as a possible 
facilitating strategy by one individual. This role of strengthening collaboration 
and building integration may be a function that is not usually included in 
describing the mission of funding agencies (see, for example, Friedman & 
Baldin, 1990), but its incorporation might result in important gains across a 
field. 



The research reported here has a number of methodological limitations. 
The interview process was often lengthy and time consuming for the 
interviewee. This was particularly true when the researchers, in several cases, 
had several extensive projects on which to report. One researcher indicated in 
a later contact that he had only reported the major variables, in spite of the 
request to identify all the variables, and that there were additional 
demographic variables that he had not discussed over the phone. An additional 
step in the procedure would require that the researchers each receive a table 



of all their responses, so that the data could be verified by them. It was 
determined that this would impose too great a burden on the researchers, who 
had been willing to donate their time for the phone interviews. The data must 
be interpreted in light of these limitations. 

Conclusions 

The results of this study suggest some future directions for the research 
community in special education and the institutional structures which support 
research. The researchers provided a number of specific suggestions for 
funding agencies and organizations to foster collaboration and improve the 
comparability of the research literature. While clearly some of these 
suggestions are already being implemented, additional support is worth 
consideration if there is to be a more integrated and meaningful research 
literature in special education. Individual autonomy and creativity should 
continue to be highly valued by the education community. Finding ways to 
work together to share information and instrumentation could, however, also 
become a valued priority. 
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Table 1 



Instrumentation (N=20 Projects) 



File data: 73 variables 

Instrumentation created for the study: 72 variables 
Published instruments: 48 variables 





Table 2 



Achievement Measures (N=20 Projects) 



Standardized Group Achievement Tests 



CBTS 2 

SAT 2 

MAT 4 

ITBS 1 

CAT 3 

State Level Tests 1 

Local Choice 1 



Individual Achievement Tests 



Bass 2 

CBM 4 

CBA 1 

WRAT (Spelling) 1 

Richardson Decoding 1 

Skill Tests 

Woodcocks- Johnson 1 

Other (e.g., grades, teacher ranking, 

criterion-referenced): 9 
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Table 3 



Classroom Observation Measures (N=20 Projects) 



CISSAR 4 

MELD 1 

Modified MELD 1 

SOBER 1 

TIES 1 
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Table 4 



Multiple Use of Published Instruments (N=20 Projects) 



Number of Instruments used in four studies: 3 
CISSAR 
MAT 
CBM 

Number of instruments used in three studies: 2 
Harter Self-Perception Scale for Children 
CAT 

Number of instruments used in two studies: 6 
Number of instruments used in one study: 23 
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Table 5 



Ranking of Reason for Selection of Research Instruments 



Created for study 

Easy to use 

Validity and reliability 

Used previously by researcher 

Requires little time 

Theoretical relationship to construct 

Instrument used in district 

Sensitive to small changes 

Need for qualitative data to compare with previous data 
as part of common data set with another research project 

Only instrument available for this variable 
Face validity for teachers 

Provides both research data and information for teachers 
Teacher administered 

Selected in collaboration with school personnel 
Required in RFP 
Curious about measure 

(N=l 8 project directors for 125 measures) 



Table 6 



Types of Technical Assistance Requested by Project Directors 



Conferences on specific topics: 



Ethnographic/qualitative research 9 

implementation 4 

Training in utilizing specific 
instrument/measurement techniques: 
Classroom observation systems 4 

Academic achievment measures 3 

Consultation 3 

Research design and analysis 3 

Sustaining innovations 2 

Adapting instruments to answer 
specific research questions 1 

Methods for measuring growth and 

change in handicapped children 1 

Discussion about appropriate 
outcome measures 1 

Procedural issues in collecting 
data in schools 1 

Sharing opportunities among researchers: 

To avoid re-inventing the wheel 3 

To provide opportunity to 
discuss problems/issues 2 

Field needs more instrumentation 
for school-based research 1 

Need for more technical assistance in 
general 1 

A lab where standardized training on 
research instruments is available 1 

Provision of "blinded" test administration, 
available on contact 1 
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Table 7 



i. D 



Forms of Researcher Collaboration 



Discussion with othe researchers 20 

Discussion with other researchers 
specifically about instrumentation 9 

Collaboration among researchers: 

Sharing instruments 10 

Common data collection 8 

Joint authorship and presentations 1 



Collaboration between researchers 
and school personnel 1 

Leadership in building 

researcher network 1 
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Table 8 



Researchers’ Suggestions to Facilitate Collaboration 

Suggestions to Funding Agencies: 

1 . Provide opportunities for collaboration early in projects 5 

2. Build consortia around common problems, with funding for collaboration 4 

3. Share written information and instruments of similar projects 2 

4. Provide incentives and contingencies for collaboration 1 

Other Suggestions: 

5. Convene conferences on topics of interest 3 

6. Utilize major conferences (such as CEC, AERA) for opportunities to meet 1 

7. Develop climate of trust, including trusted leadership 4 

8. Joint authorships, with senior authorship rotated 1 

9. Consider time pressures on researchers . 1 



Table 9 



Additional Issues/Concems Raised by Researchers 



1. Collaboration should not be forced (2) 

2 . There is a need for more theoretical discussions in the field (1) 

3. There needs to be more tolerance for divergent approaches (such as quantitative research) by funding 
agencies (2) 

4. There needs to be recognition of how "needy" schools are; asking schools to be collaborative in research, 
given their own needs, is inappropriate. Funds should be built in to reward schools for participation in 
research (1) 
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October 31, 1989 



A F1 A 
Re: A F2 A 
Dear A F3 A , 

This is a follow-up to a letter that we sent to you last year regarding 
the above-named project, which is funded by the Office of Special Education 
Programs (OSERS) to examine a range of instructional and administrative issues 
in educating learning disabled and mildly handicapped students in general 
education classrooms. We were not able to schedule an interview with you at 
that time, but would now like to arrange a time to speak with you about this 
project. 

As we indicated in the previous letter, the Temple University Center for 
Research in Human Development and Education was funded to synthesize the 
state of the field and the literature on important variables that influence 
student learning. To this end, we are gathering information on research 
methodology and research development across projects relating to these 
variables. We are now in the final phases of data collection for this project, 
and wish to arrange for your participation. 

During the next month, Carol Lidz and I will conduct telephone 
interviews with the researchers from the OSERS-funded projects who have not 
yet been ccracted to learn about the research questions and methodologies, 
data collection plans, and activities across the states. These interviews take 
approximately one hour. At the conclusion of this process, all project 
researchers who express an interest will receive a brief report of our findings. 

There are several appointment times, mostly on Monday and Wednesday 
mornings, that are available throughout November. Veronica Norris from our 
office will be calling you in the next few days to schedule a time that is 
mutually convenient. If any of the prescheduled times is not convenient, 
please indicate to her the best times to reach you. 

We are aware that your projects are in various stages of implementation, 
and that some of you will answering interview questions either prospectively or 
retrospectively. Either point of view is fine. In brief, we will be asking you 
about the following: 

-or. 1 what variables have you, or are you planning on, 
collecting your data? 



- 2 - 



•what instruments are you or are you planning on using? 

-why did you choose these particular instruments? 

-have there been any barriers to data collection, or do 
you anticipate any? 

In addition to these questions, we can include any issues concerning 
instrumentation that you may wish to discuss. 

Thank you for your interest and willingness to participate, we look 
forward to spending time on the telephone with you during the next month. 

Sincerely, 



Sylvia Rosenfield 

Senior Research Associate 

Professor of School Psychology 



SR/ab 
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Synthesis Interview Protocol 
Research Instrumentation 



Name of Interviewer: 
Date of Interview: 



Introduction: We appreciate you giving us this 

synthesis project is looking at the methodologies used 
the major researchers in terms of instrumentation, 
make this as brief and as interesting as possible, 

taping and taking notes. 



time. 




The 


by 


some 


of 


We 


hope 


to 


We 


will 


be 



1. Project Name: 



2 . Project Director: 



3. Project Personnel Interviewed (Name and Title): 



4. Date of Interview: 



5. Phone Number: 

BEST COPY AVAILABLE 

6. Address: 





7. Dates of Project: 



S. Geographic Location: 



For Each Variable, collect information on items 9-16. Refer 

Logistics in Practice form.. 



17. What kind(s) of technical assistance with respect 
instrumentation would you find helpful? 

Are you interested in conferences: 

On particular topics: 
consul tat ion /implementation 
ethnographic/qualitative data? 

Other (specify): 



Who would you like to have attend/present? 



Other: 



IS. What kinds of collaboration with other researchers are 
engaged in related to this project. 

Sharing instruments 

Discussion about: 

Literature 

Findings 

Instruments 

Data Analysis 
Common Data Collection 

Joint Presentation of Findings 

Other (Specify): 



1 



> O 




to 



to 



you 



34 



19. 


What 


do 


you 


believe would lead to 


increased 


the 


use 


of 


similar 


instrumentation and/or 


common 



among researchers in this field? 



collaboration on 
data collection 



20. Additional Comments: 
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Instrumentation Survey 
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APPENDIX B 

INFLUENTIAL VARIABLES IN LEARNING INCLUDED IN 
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I: State and District Variables 



Number 
of Studies 



District Level Demographics and Mar ker Variables 
School district size 

Degree of school district bureaucratization 

Degree of school district centralization 

Presence of contractual limits on after school meetings 

Presence of contractual limits on class size 

Presence of contractual restrictions on activities 
performed by aides 

Degree of central office assistance and support for programs 
Degree of board of education support for instructional programs 
Per pupil expenditure 
Efficiency of transportation system 
State Level Policy Variables 

Teacher licensure requirements 

Degree of state control over textbooks 

Degree of state control over curriculum 

Academic course and unit requirements 

Minimum competency testing requirements 

Adherence to least restrictive environmeni/mainstreaming 
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n: Out of School Context ual Variables 



Communit y Variables 

Socioeconomic level of community 2 

Ethnic mix of community 

Quality of social services for students 

Rural/Urban/Suburban n 

Peer Group Variables 

Level of peers' academic aspirations 

Level of peers' occupational aspirations 

Presence of well defined clique structure 

Degree of peers’ substance abuse 

Degree of peers' criminal activity 

Home Environment and Parental Support Variables 

Educational environment (e.g. number of books and magazines 

in home) 

Parental involvement in assuring completion of homework 

Parental involvement in assuring regular school attendance 

Parental monitoring of student television viewing 1 

Parental participation in school conferences and l 

related activities 

Parental application of appropriate, consistent discipline 

Parental expressions of affection to children 



Parental interest in student's school work 1 

Parental expectation for academic success 

Parental Involvement 2 

Parental Satisfaction with School Program 2 

Family Constellation _J_ 




Student Use of Out of School Tune Variables 



Student participation in dubs and extracurricular 
school activities 

Amount of time spent on homework 

Amount of time spent on leisure reading 

Amount of time spent viewing educational television 

Amount of time spent viewing noneducational television 



m: School Level Variables 



Demographic and Marker Variables 
Public versus private school 
Size of school 

Level of Chapter I (compensatory education) funding 

Level of Title VD (bilingual) funding 

Level of PL 94- 142 (handicapped) funding 

Mix of socioeconomic levels in the school 

Mix of cultural/ethnic groups in the school 

Mix of student language backgrounds in the school 

Teacher age 



Teacher degree 
Teacher experience 



Teacher competence 

Teacher/Administrator Dec ison Making Variables 

Teacher and administrator consensus on school values, 
norms, and roles 

Principal actively concerned with instructional program 

Teacher involvement in curricular decision making 

Teacher involvement in instructional decision making 

Teacher involvement in resource allocation decisions 

Teacher involvement in finding ways to increase academic 
performance 



Teacher confidence in handling problems 



School Culture Variables (E thos Conducive to Teaching and 1 ^arninci 
Use of cooperative, not exclusively competitive, goal structures 



School-wide emphasis on and recognition of academic 
achievement 

Low staff absenteeism 

Low staff turnover 

Low stalT alienation 

Active collaboration between regular classroom teachers and 
special education teachers 

Safe, orderly school climate 

Degree of school personnel professional collaboration 

Rate of referrals to special ed 

Staff satisfaction 

Concerns regarding change 

Teacher expeciabons/toJerance for student behavior 

School-Wide Policy and O perational Variables 

Presence of "effective schools program" 

Explicit school grading and academic progress policies 

Explicit school-* ide discipline policy 

Explicit school-wide attendance policy 

Coordination of pullout programs for handicapped students 
with regular instructional programs 

Use of multi-age grouping 

Use of instructional teaming 

Use of cross-age tutoring 

Use of peer tutoring 

Use of academic tracking for specific school 
subject areas 

Minimization of external classroom disruptions 
(e.g., broadcast announcements) 





Adherence to least restrictive environment/mainstreaming 
Minimum use of suspension and expulsion as discipline tools 
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Accessibility Variables 



Accessibility of educational program (overcoming 

architectural, communication, and environmental barriers) 

Parental Involvement Policy Variables 

Parental involvement in improvement and operation of 
instructional programs 

School-sponsored parenting skills workshops (e.g., behavior 
modification. Parent Effectiveness Training) 



IV: Student Variables 

Demographic and Marker Variables 

Chronological age 6 

Socioeconomic status 12 

Gender 14 

Ethnicity 10 

Fust or native language ^ 

Physical and health status 

Special education classifications (e.g., EMR, LD) 16 

Grade level 15 

At risk ]_ 

Birthorder j_ 

History of Educational Placements 

Prior grade retentions 

Prior special placements 

Current placement in regular class versus self-contained 2_ 

special education class 

Age/Grade match ]_ 

Social and Behavioral Variables 3 

Positive, nondisruptive behavior 

Appropriate activity level 1_ 

Cooperativeness with teacher 3_ 

Cooperativeness with peers ±_ 

Ability to make friends with peers 2_ 




Motivational and Affective Variables 



Attitude toward school 

Attitude toward teachers 

Attitude toward subject matter instructed 

Motivation for continual learning 

Independence as a learner 

Perseverance on learning tasks 

Self-confidence 

Academic self-competence concept in subject area instructed 
Attributions for success and failure in subject area instructed 
Self-concept 
Cognitive Variables 

Piagetian stage of cognitive development 
Level of reasoning (fluid) ability 
Level of spatial ability 
Memory 

Level of general academic (crystallized) knowledge 

Level of specific academic knowledge in subject area instructed 

Level of reading comprehension ability 

Level of writing ability 

Level of computational ability 

Level of oral fluency 

Level of listening skills 

Learning styles (e.g., field independent, visual/auditory learners, 
high cognitive complexity) 

Level of spelling 

IQ 




Learning potential 
Language level 



Meiacognitive Variables 



Self-regulatory, self-conuol strategies (e.g., control of 
attention) 

Comprehension monitoring (planning; monitoring effectiveness 
of attempted actions; monitoring outcomes of actions; 
testing, revising, and evaluating learning strategies) 

Positive strategies for coping with failure 

Positive strategies to facilitate generalization of concepts 

Psvchomotor Variables 

Psychomotor skills specific to area instructed 



V: Program Design Variables 



Democraphic and Marker Variables 

Size of instructional group (whole class, small group, 
one-on-one instruction) 

Proportion of students with special needs served in regular classes 

Number of classroom aides required 

Resources needed 

Curriculum and Instructi onal Variables 

Clearly presented academic, social, and altitudinal program 
goals/outcomes 

Use of explicit goal/objective setting for instruction of 
individual student (e.g.. Individualized Educational 
Plans [XEPs]) 

Use of mastery learning techniques, including use of 

instructional cues, engagement, and corrective feedback 

Use of cooperative learning strategies 

Use of personalized instructional program 

Use of prescriptive instruction combined with aspects of 
informal or open education 

Use of diagnostic-prescriptive methods 
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Curriculum and Instruction al Variables (continued 
Use of computer-assisted instruction 



Use of crisis management techniques to control classroom 
disrupt! veness 

Use of program strategies for favorable affective climate 

Alignment among goals, contents, instruction, assignments and 
evaluation 

Curniculum units integrated around key discipline-based 
concepts 

Use of multidisciplinary approaches to instructional planning 
(including diagnosis in educational planning) 

Presence of information in the curriculum on individual 
differences and commonalities (including handicapping 
conditions) 

Presence of culturally diverse materials in the curriculum 
Curriculum Design Variables 

Materials employ alternative modes of representation 

Material is presented in a cognitively efficient manner 

Materials employ explicit and specific objectives 

Materials employ advance organizers 

Materials employ learning heirarchies 

Materials are tied to assessment and diagnostic tests 

Availability of materials and activities prepared specifically 

for use with whole classroom, small groups, or one-on-one 
instruction 

Degree of structure in curriculum accommodates needs of 
different learners 

Student interests guide selection of a significant portion of 
content 



Availability of materials and activities for students with 
different abilities 



Curriculum Design Variables (continued) 



Availability of materials and activities for students with 
different learning styles 

Developmental issues considered 

Student experiences considered 



VI: Implementation. Clas sroom Instruc tion, and Climate Variables 
Classroom Implementatio n Support Variables 

Creation and maintenance of necessary instructional 
materials 

Adequacy in the configuration of classroom space 

Availability of classroom aides 

Use of written records to monitor student progress 

Establishing efficient classroom routines and communicating 
rules and procedures 

Developing student self-responsibility for independent study 
and planning of one's own learning activities 

Classroom I nstructional Variables 

Prescribing individualized instruction based on perceived match 
of type of learning tasks to student characteristics (e.g., 
ability, learning style) 

Use of procedures requiring rehearsal and elaboration of new 
concepts 

Use of clear and organized direct instruction 

Systematic sequencing of instructional events and activities 

Explicit reliance on individualized educational plans (LLPs) in 
planning day-to-day instruction for individual students 

Use of instruction to surface and confront student 
misconceptions 

Use of advance organizers, overviews, and reviews of objectives 
to structure information 

Clear signalling of transitions as the lesson progresses 
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Classroom Instructional Variables (c ontinued) 



Significaiu redundancy in presentation of content 

Teacher conveys enthusiasm about the content 

Directing students' attention to the content 

Using reinforcement contingencies 1 

Setting and maintaining clear expectations of content mastery 1 

Providing frequent feedback to students about their performance L 

Explicitly promoting effective metacognidve learning strategies 

Promoting learning through student collaboration (e.g., peer L 

tutoring, group work) 

Corrective feedback in event of student error 



Flexible grouping that enables students to work to improve 

and change status/groups 

Teaching for meaningful understanding 

Degree to which student inquiry is fostered 

Scaffolding and gradual transfer of responsibility from 

teacher to student 

Degree to which assessment is linked with instruction 

Skills taught within the context of meaningful application 

Good examples and analogies to concretize the abstract and 

familiarize the strange 

Consideration of the teacher’s use of language in the instructional L 

process 

Explicitly promoting student self-monitoring of comprehension 

Quantity oflnstruetion Variables L 

Length of school year 

Length of school day 

Time on task (amount of time students are actively engaged in & 

learning) 




Quantity of Instruction V ariables (continued) 



Time spent in direct instruction on basic skills in reading 

Time spent in direct instruction on basic skills in mathematics 

Time allocated to basic skills instruction by regular classroom 
teacher 

Time allocated to basic skills instruction by special education 
teacher 

Difference between academic learning time and allocated 
learning time 

Time spent out of school on homework 

Time spent out of school viewing educational television 

Time spent out of school in informal learning experiences (e.g., 
museum trips, scouts) 

Nature of regular classroom content missed by students during 



participation in pullout programs 

Attendance L 

Tardiness L 

Qassroom A ssessment Variables 

Use of assessments to create detailed learner profiles rather L 

than simple classifications or unelaborated total scores 

Use of assessment as a frequent, integral component of L 

instruction 

Accurate, frequent measurement of basic skills in reading 



Accurate, frequent measurement of basic skills in mathematics 
Classroom M anagement Variables 

Minimal disruptiveness in classroom (e.g., no excessive noise 
no students out of place during instructional activities, 
no destructive activities) 

Group alerting (teacher uses questioning/recitation strategies 
that maintain active participation by all students) 

Learner accountability (teacher maintains student awareness 
of teaming goals and expectations) 

Transitions (teacher avoids disruptions of learning 
activities, brings activities to a clear and natural 
close, and smoothly initiates new activity) 




p; BEST COPY AVAILABLE 



49 



Classroom Management Va riables (continued) 



Teacher "wiihitness” (teacher is continually aware of events and activities 
and minimizes disruptiveness by timely and nonconfrontational 
actions) 

Student and Teacher Interactions: Social Variables 

Student initiates positive verbal interactions with other 
students and with teachers 

Student responds positively to questions from other students 
and from teacher 

Teacher reacts appropriately to correct and incorrect answers 

Teacher reinforces positive social interactions with students 
rejected by peers 

Teacher provides explicit coaching on appropriate social 
beheviors 

Teacher provides explicit coaching to reduce aggression 

Student and Teacher Interactions: Academic Variables 

Teacher asks academic questions frequently 

Teacher asks questions predominantly low in difficulty 

Teacher asks questions that are predominantly low in 
cognitive level 

Teacher maintains high post-question wait time 

Frequent calls for extended, substantive oral and written 
response (not one-word answers) 

Classroom Climate Variables 

Cohesiveness (members of class are friends sharing common 
interests and values and emphasizing cooperative goals) 

Low friction (students and teacher interact in a 

considerate and cooperative way, with minimal 
abrasiveness) 

Low cliqueness (students work with many different classmates, 
and not just with a few close friends) 

Satisfaction (students are satisfied with class activities) 

Speed (the pacing of instruction is appropriate for the majority 
of the students) 



Task difficulty (students are continually and appropriately 
challenged) 



Classroom Climate Variables (continued) 

Low apathy (class members are concerned and interested in 
what goes on in the class) 

Low favoritism (all students are treated equally well in the 
class, and given equal opportunities to participate) 

Formality (students are asked to follow explicitly stated rules 
concerning classroom conduct and activities) 

Goal direction (objectives of learning activities are specific 
and explicit) 

Democracy (all students are explicitly involved in making some 
types of classroom decisions) 

Organization (class is well organized and well planned) 

Diversity (the class divides its efforts among several different 
purposes) 

Environment (needed or desired books and equipment are 
readjly available to students in the classroom) 

Competition (students compete to see who can do the best work) 
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